Papers by Jackie Chi Kit Cheung

A Cross-Domain Transferable Neural Coherence Model (P19-1)

Challenge:	Existing REG systems rely on entity-specific supervised training, which means they cannot handle entities not seen during training.
Approach:	They propose a deep neural network model that encodes both the local context and an external profile of the entity to generate reference realizations.
Outcome:	The proposed model outperforms baselines on three different splits of the WebNLG dataset according to automatic and human evaluations.

Copied to clipboard

Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, Jackie Chi Kit Cheung

Challenge:	Existing coherence models do not generalize to unseen categories of text . previous work advocates for generative models for cross-domain generalization .
Approach:	They propose a local discriminative neural model with a smaller negative sampling space that can discriminate against incorrect orderings.
Outcome:	The proposed model outperforms state-of-the-art methods on a standard benchmark dataset on the Wall Street Journal corpus and multiple challenging settings on Wikipedia articles.

The Topic Confusion Task: A Novel Evaluation Scenario for Authorship Attribution (2021.findings-emnlp)

Copied to clipboard

Malik Altakrori, Jackie Chi Kit Cheung, Benjamin C. M. Fung

Discourse-Aware Unsupervised Summarization for Long Scientific Documents (2021.eacl-main)

Challenge:	Autorship attribution is the problem of identifying the most plausible author of an anonymous text from a set of candidate authors.
Approach:	They propose a topic confusion task where they switch the author-topic configuration between training and testing sets and propose attribution errors that are caused by the topic shift and by the features’ inability to capture the writing styles.
Outcome:	The proposed task combines author-topic configuration with other features to lower topic confusion and higher attribution accuracy.

Copied to clipboard

Yue Dong, Andrei Mircea, Jackie Chi Kit Cheung

A Controlled Reevaluation of Coreference Resolution Models (2024.lrec-main)

Challenge:	Existing extractive models for short news summarization are weak, despite recent advances in abstractive summarizing.
Approach:	They propose an unsupervised graph-based ranking model that uses a hierarchical graph representation to determine sentence importance.
Outcome:	The proposed model outperforms strong unsupervised baselines by wide margins in automatic metrics and human evaluation.

Copied to clipboard

Ian Porada, Xiyuan Zou, Jackie Chi Kit Cheung

The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution (P19-1)

Challenge:	a pretrained language model is used in state-of-the-art coreference resolution models.
Approach:	They evaluate five coreference resolution models and control for language model used . they find that encoder-based CR models outperform decoder--based models in accuracy .
Outcome:	The encoder-based model outperforms the decoder--based models in accuracy and speed . older model generalizes the best to out-of-domain textual genres .

Copied to clipboard

Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, Jackie Chi Kit Cheung

A Knowledge Hunting Framework for Common Sense Reasoning (D18-1)

Challenge:	Existing methods for coreference resolution exploit the number and gender of antecedents or have been handcrafted and do not reflect the diversity of naturally occurring text.
Approach:	They propose a trick to improve resolution by antecedent switching to target common-sense understanding and world knowledge.
Outcome:	The proposed method achieves state-of-the-art results on the GAP coreference task.

Copied to clipboard

Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung

Factual Error Correction for Abstractive Summarization Models (2020.emnlp-main)

Challenge:	a new system that uses common sense to solve a common sense problem is developed . a winograd schema challenge and a choice of plausible alternatives are popular tests .
Approach:	They propose an automatic system that achieves state-of-the-art results on the Winograd Schema Challenge . they use a knowledge hunting module to gather web text for problem resolutions .
Outcome:	The proposed system achieves state-of-the-art on the Winograd Schema Challenge . it improves F1 performance on the full WSC by 0.21 over the previous best .

Copied to clipboard

Meng Cao, Yue Dong, Jiapeng Wu, Jackie Chi Kit Cheung

Summarization

Challenge:	Existing methods for abstractive summarization are unable to ensure factual consistency of generated summaries.
Approach:	They propose a post-editing corrector module to identify and correct factual errors in generated summaries.
Outcome:	The proposed model outperforms existing models on CNN/DailyMail dataset on factual consistency evaluation.

The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources (2023.acl-long)

Copied to clipboard

Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung

Textual Time Travel: A Temporally Informed Approach to Theory of Mind (2021.findings-emnlp)

Challenge:	Existing models that make inferences using information from multiple sources are largely understudied .
Approach:	They propose a test suite of coreference resolution subtasks that require reasoning over multiple facts and introduce subtask where knowledge is present only at inference time using fictional knowledge.
Outcome:	The proposed subtasks differ in terms of which knowledge sources contain the relevant facts and where knowledge is present only at inference time using fictional knowledge.

Copied to clipboard

Akshatha Arodi, Jackie Chi Kit Cheung

Linguistic Theories

Challenge:	Existing models that attribute mental states to oneself and others perform poorly on false belief tasks where beliefs differ from reality.
Approach:	They propose a temporally informed approach for improving the theory of mind capability of memory-augmented neural models by integrating priors about entities’ minds and tracking their mental states over time through an extended passage.
Outcome:	The proposed model improves performance on false belief tasks where beliefs differ from reality, especially when the dataset contains distracting sentences.

Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples (N19-1)

Copied to clipboard

Krtin Kumar, Jackie Chi Kit Cheung

Challenge:	Neural abstractive summarization systems generate summary texts conditioned on the input source text, and have recently achieved high ROUGE scores on benchmark summarizing datasets.
Approach:	They propose to analyze existing neural abstractive summarization systems by comparing their performance to human-written summaries.
Outcome:	The proposed systems perform better than human-written summarizations on different datasets and show that they are able to understand deeper syntactic and semantic structures.

Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses (D19-1)

Copied to clipboard

Matt Grenander, Yue Dong, Jackie Chi Kit Cheung, Annie Louis

Summarization

Challenge:	Sentence position is a strong feature for news summarization, since the lead often summarizes the key points of the article.
Approach:	They propose two techniques to make neural systems sensitive to the importance of content in different parts of the article by using random shuffled sentences to pretrain the model.
Outcome:	The proposed techniques improve the performance of a competitive reinforcement learning based extractive system, with the auxiliary loss being more powerful than pretraining.

Active Learning with Non-Uniform Costs for African Natural Language Processing (2026.findings-eacl)

Copied to clipboard

Bonaventure F. P. Dossou, Ines Arous, Audrey Durand, Jackie Chi Kit Cheung

Responsible & Trustworthy NLP

Challenge:	Annotating datasets for African languages is challenging due to the continent's vast linguistic diversity, complicating development of NLP systems.
Approach:	They propose a cost-aware active learning method that integrates BatchBALD acquisition strategy with a 0-1 Knapsack optimization objective to select informative and budget-efficient samples.
Outcome:	The proposed method outperforms BALD, BatchBALD, and stochastic sampling variants across cost scenarios on the MasakhaNEWS multilingual news classification benchmark covering 11 African languages.

A Generalized Knowledge Hunting Framework for the Winograd Schema Challenge (N18-4)

Copied to clipboard

Ali Emami, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung

How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG (D19-1)

Challenge:	a new system that performs well on common-sense reasoning tasks is developed . the Winograd Schema Challenge (WSC) is a popular alternative to the Turing test .
Approach:	They propose an automatic system that performs well on two common-sense reasoning tasks.
Outcome:	The proposed system improves performance on the Winograd Schema Challenge and COPA by 0.16 over the previous best.

Copied to clipboard

Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung

TESA: A Task in Entity Semantic Aggregation for Abstractive Summarization (2020.emnlp-main)

Challenge:	a recent study has improved the state-of-the-art on common-sense reasoning benchmarks . a san francisco-based approach to common-ense reasoning is challenging .
Approach:	They propose to use common-sense reasoning benchmarks to test machine learning's common-sentence inference task SWAG to test common-mind systems.
Outcome:	a new study shows that improved performance on common-sense reasoning benchmarks is genuine . the proposed task is more difficult than the current one, but it is more efficient than the previous one.

Copied to clipboard

Clément Jumel, Annie Louis, Jackie Chi Kit Cheung

Post-Editing Extractive Summaries by Definiteness Prediction (2021.findings-emnlp)

Challenge:	Abstractive summarization systems focus on paraphrasing and simplifying the source content, to the exclusion of such semantic abstraction capabilities.
Approach:	They propose a dataset and task to fine tune an abstractive summarization model to generate aggregations of 5.3K entities from a crowd-sourced dataset.
Outcome:	The proposed task and dataset show that the proposed model can generate aggregations at a semantic level, but that it is too complex to use.

Copied to clipboard

Jad Kabbara, Jackie Chi Kit Cheung

A Hierarchical Neural Attention-based Text Classifier (D18-1)

Challenge:	Abstract: Extractive summarization has been the mainstay of automatic summarizing for decades, but it still suffers from coreference issues arising from extracting sentences away from their original context.
Approach:	They propose a post-editing step that generates linguistic decisions that lead to improved extractive summaries by predicting definiteness of noun phrases.
Outcome:	The proposed system generates linguistic decisions that improve the quality of the extractive summaries.

Copied to clipboard

Koustuv Sinha, Yue Dong, Jackie Chi Kit Cheung, Derek Ruths

On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020.starsem-1)

Challenge:	Existing hierarchical classification models are unable to handle large corpora and the number of categories increases with increasing corpus.
Approach:	They propose to use external knowledge to introduce a hierarchical neural attention-based classifier to help with the classification of documents.
Outcome:	The proposed model performs better than or comparable to state-of-the-art hierarchical models at significantly lower computational cost while maintaining high interpretability.

Copied to clipboard

Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung

Modeling Event Plausibility with Consistent Conceptual Abstraction (2021.naacl-main)

Challenge:	Existing studies have found that BERT can correctly retrieve noun hypernyms in cloze tasks, but this does not correspond to systematic knowledge in BERT.
Approach:	They propose to use BERT to probe for hypernymy knowledge encoded in representations for cloze tasks to find out whether it is systematic or not .
Outcome:	The proposed model can retrieve hypernyms in cloze tasks, but not systematic knowledge in BERT.

Copied to clipboard

Ian Porada, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung

An Analysis of Dataset Overlap on Winograd-Style Tasks (2020.coling-main)

Challenge:	Understanding natural language requires common sense, one aspect of which is the ability to discern the plausibility of events.
Approach:	They propose a method of forcing model consistency that improves correlation with human plausibility judgements.
Outcome:	The proposed method improves correlation with human plausibility judgements.

Copied to clipboard

Ali Emami, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung

EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing (P19-1)

Challenge:	a large number of test instances overlap considerably with pretraining corpora, a study finds . for a number of years, models struggled to exceed chance-level performance .
Approach:	They analyze the effects of varying degrees of overlaps that occur between pretraining corpora and test instances in WSC-style tasks.
Outcome:	The WSC-Web dataset is the largest to date and has lower overlaps with current pretraining corpora.

Copied to clipboard

Yue Dong, Zichao Li, Mehdi Rezagholizadeh, Jackie Chi Kit Cheung

TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion (2020.emnlp-main)

Challenge:	Current sentence simplification systems are variants of sequence-to-sequence models adopted from machine translation.
Approach:	They propose a sentence simplification model that learns explicit edit operations via a neural programmer-interpreter approach.
Outcome:	The proposed model outperforms state-of-the-art models on three benchmark text simplification corpora in terms of SARI (+0.95 WikiLarge, +1.89 WikiSmall, -1.41 Newsela)

Copied to clipboard

Jiapeng Wu, Meng Cao, Jackie Chi Kit Cheung, William L. Hamilton

Multimodality

Challenge:	Existing methods for static knowledge graphs do not explicitly leverage multi-hop structural information and temporal facts from recent time steps to enhance their predictions.
Approach:	They propose a framework to leverage time-dependent temporal information to infer missing facts in temporal knowledge graphs.
Outcome:	The proposed framework achieves 10.7% improvement in Hits@10 across three standard benchmarks.

Deconstructing word embedding algorithms (2020.emnlp-main)

Copied to clipboard

Kian Kenyon-Dean, Edward Newell, Jackie Chi Kit Cheung

Representation Learning

Challenge:	Word embeddings are reliable feature representations of words used in many NLP tasks today.
Approach:	They propose to deconstruct Word2vec, GloVe and others into a common form . they propose to generalize several word embedding algorithms into . a low rank embedder framework is proposed to generalise the algorithms into one common form.
Outcome:	The proposed framework can be used to make word embeddings more performant.

Responsible AI Considerations in Text Summarization Research: A Review of Current Practices (2023.findings-emnlp)

Copied to clipboard

Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler

On-the-Fly Attention Modulation for Neural Generation (2021.findings-acl)

Challenge:	a recent study examines research and reporting practices for text summarization tasks . text summaries are often overlooked by the responsible AI community .
Approach:	They examine research and reporting practices in the context of text summarization . they find that relatively few papers engage with possible stakeholders .
Outcome:	The findings highlight current research practices and provide recommendations on research directions.

Copied to clipboard

Yue Dong, Chandra Bhagavatula, Ximing Lu, Jena D. Hwang, Antoine Bosselut, Jackie Chi Kit Cheung, Yejin Choi

Learning with Rejection for Abstractive Text Summarization (2022.emnlp-main)

Challenge:	Degeneration of neural text is associated with insufficient learning of task-specific characteristics by the attention mechanism.
Approach:	They propose to use attention modulation to inject priors into inference to improve fluency, creativity, and commonsense reasoning in neural text generation models.
Outcome:	The proposed method improves fluency, creativity, and commonsense reasoning, and significantly reduces sentence-level repetition.

Copied to clipboard

Meng Cao, Yue Dong, Jingyi He, Jackie Chi Kit Cheung

Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences (2022.coling-1)

Challenge:	Existing abstractive summarization systems produce non-factual summaries due to noise in the training dataset.
Approach:	They propose a training objective for abstractive summarization based on rejection learning that learns whether or not to reject potentially noisy tokens.
Outcome:	The proposed method significantly improves the factuality of generated summaries in automatic and human evaluations when compared to baseline models.

Copied to clipboard

Jad Kabbara, Jackie Chi Kit Cheung

Learning Efficient Task-Specific Meta-Embeddings with Word Prisms (2020.coling-main)

Challenge:	Presuppositions are assumptions that are taken for granted by an utterance.
Approach:	They propose to use heuristics to create alternative "contrastive" test cases . they also analyze samples from ImpPres datasets to better understand their predictions .
Outcome:	The proposed model performs better on the ImpPres dataset than on the other datasets.

Copied to clipboard

Jingyi He, Kc Tsiolis, Kian Kenyon-Dean, Jackie Chi Kit Cheung

Human-Centered Evaluation of Language Technologies (2024.emnlp-tutorials)

Challenge:	Word embeddings possess different lexical properties depending on the notion of context defined at training time.
Approach:	They introduce a meta-embedding method that learns to combine source embeddings according to the task at hand.
Outcome:	The proposed method improves performance on six extrinsic evaluations over other methods.

Copied to clipboard

Su Lin Blodgett, Jackie Chi Kit Cheung, Vera Liao, Ziang Xiao

Challenge:	a lack of human-centered considerations about people’s needs for language technologies is causing an “evaluation crisis” in NLP.
Approach:	This tutorial introduces perspectives and methodologies from human-computer interaction (HCI) it will introduce what to evaluate for, how generalizable the results are to the real-world contexts, and pragmatic costs to conduct the evaluation.
Outcome:	This tutorial introduces perspectives and methodologies from human-computer interaction (HCI) the tutorial will also encourage reflection on how these HCI perspectives and methods can complement NLP evaluation through Q&A discussions and a hands-on exercise.

ADEPT: An Adjective-Dependent Plausibility Task (2021.acl-long)

Copied to clipboard

Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung

Can a Gorilla Ride a Camel? Learning Semantic Plausibility from Text (D19-60)

Challenge:	ADEPT is a large-scale semantic plausibility task that requires a significant degree of world knowledge and common-sense reasoning.
Approach:	They propose a large-scale semantic plausibility task that pairs 16 thousand sentences with slightly modified versions obtained by adding an adjective to a noun.
Outcome:	The proposed task is easier for humans (85% accuracy), but more difficult for transformer-based models (71% accuracy).

Copied to clipboard

Ian Porada, Kaheer Suleman, Jackie Chi Kit Cheung

Varta: A Large-Scale Headline-Generation Dataset for Indic Languages (2023.findings-acl)

Challenge:	Existing work on modeling semantic plausibility has focused on physical plausability but distributional methods fail when tested in supervised settings.
Approach:	They propose to use large pretrained language models to model plausibility in supervised settings by extracting attested events from a large corpus and injecting explicit commonsense knowledge into a distributional model.
Outcome:	The proposed model is effective in modeling plausibility in a supervised setting.

Copied to clipboard

Rahul Aralikatte, Ziling Cheng, Sumanth Doddapaneni, Jackie Chi Kit Cheung

BanditSum: Extractive Summarization as a Contextual Bandit (D18-1)

Challenge:	Varta dataset includes more than 41 million pairs of headlines and articles in 14 different Indic languages (and English)
Approach:	They present a large-scale multilingual dataset for headline generation in Indic languages.
Outcome:	The Varta dataset includes more than 41 million pairs of headlines and articles in 14 different Indic languages (and English) the data can be used to train strong language models that outperform competitive baselines in both NLU and NLG benchmarks.

Copied to clipboard

Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, Jackie Chi Kit Cheung

Multi-Fact Correction in Abstractive Text Summarization (2020.emnlp-main)

Challenge:	Existing methods for extractive summarization are heuristically generated and require a set of binary labels to be selected.
Approach:	They propose a method for training neural networks to perform single-document extractive summarization without heuristically-generated extractive labels.
Outcome:	The proposed method achieves better ROUGE scores than the state-of-the-art methods and significantly fewer update steps than competing approaches.

Copied to clipboard

Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, Jingjing Liu

Source-summary Entity Aggregation in Abstractive Summarization (2022.coling-1)

Challenge:	Existing abstractive summarization systems generate incorrect facts with respect to the source text.
Approach:	They propose a suite of two factual correction models that leverages question-answering knowledge to make corrections in system-generated summaries via span selection.
Outcome:	The proposed model improves factuality of news summarization without sacrificing summary quality.

Copied to clipboard

José Ángel González, Annie Louis, Jackie Chi Kit Cheung